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ABSTRACT 
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of survey administration. The implications of the new ability to track 
response behavior patterns will have a major impact on the field of survey 
research. This study explored these patterns with regard to the communication 
of follow-up reminders and the sensitivity to survey content. The study 
examined the response rate and the response pattern of an electronic survey 
distributed to program directors of postgraduate medical education programs 
in Canada with regard to the adoption of the CanMEDs roles as developed by 
the Royal College of Physicians and Surgeons of Canada. Within a 3 -week 
period, the survey response rate was 63% (356 people) , similar to those 
expected of pencil -and-paper surveys for this professional group (J. Boser 
and K. Green, 1997) . It was determined that there was not a significant 
difference in the nature or rate of responses before and after a follow-up 
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the familiarity or support of the survey content. An appendix contains the 
follow-up letter. (SLD) 
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Abstract 

The advent of the World Wide Web as a platform for the 
dissemination of survey instruments to diverse populations and the 
electronic recording of response data has changed the timeliness 
and cost effectiveness of survey administration. The implications 
of the new ability to track response behavior patterns will have a 
major impact on the field of survey research. This study explores 
these patterns specifically with regard to the communication of 
follow-up reminders and the sensitivity to survey content. 

This study examined the response rate and the response 
pattern of an electronic survey distributed to program directors of 
post-graduate medical education programs in Canada with regard 
to the adoption of the CanMEDS roles as developed by the Royal 
College of Physicians and Surgeons of Canada. Within a three- 
week period, the survey response rate was 63% - similar to those 
expected of pencil-and-paper surveys for this professional group 
(Boser and Green, 1997). It was determined that there was not a 
significant difference in the nature or rate of responses before and 
after a follow-up reminder. Furthermore, the rate and nature of 
responses were not dependent on the familiarity or support of the 
survey content. 
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A. Objectives 

The advent of the World Wide Web as a platform for the dissemination of survey 
instruments to diverse populations and the electronic recording of response data has changed the 
timeliness and cost effectiveness of survey administration. The implications of the new ability to 
track response behavior patterns will have a major impact on the field of survey research. This 
study explores these patterns specifically with regard to the communication of follow-up 
reminders and the sensitivity to survey content. 

The Royal College of Physicians and Surgeons of Canada (RCPSC) employed a web- 
based survey for a formative program evaluation of the new framework for medical specialist 
education called the CanMEDS roles (RCPSC, 2000). Since the inception of these roles in 1996, 
medical educators across the 16 medical schools across Canada including 53 medical specialties 
and sub-specialties have been incorporating this framework into their objectives of training, their 
in-training evaluations and their examinations. In order to study the process of implementing 
these CanMEDS roles, the Office of Education of the Royal College commissioned a study to 
examine the progress of the integration of these roles into the education and evaluation of 
medical residents. As part of this study, this survey was administered in English or French to 
medical educators across all medical specialties for all of Canada in March 2001. 

Researchers at the Royal College investigated the response pattern in order to optimize 
future survey administration. To this end, the objectives of this paper are to: 

i) Describe the development and rationale for using a web-based survey methodology for 
the assessment of the opinions of a national diverse group such as key Canadian medical 
educators. 

ii) Analyze responses over time as per the application of Dillman's (2000) principles of 
survey implementation, specifically, to report the efficacy of sending a follow-up 
reminder notice to respondents and the nature of the survey participants that respond to 
the follow-up letter. 

It was expected that follow-up contact with respondents would increase the response rate 
at a rate similar to mail surveys (Boser, 1996). 



iii) In addition to examining the changes in response rate, an analysis was made to 
examine the nature of the responses based on the respondents' report of familiarity and 
favourability with the CanMEDS roles as well as the nature of their open-ended 
comments. 



B. Perspectives 



The Use of Web-surveys 

The use of the Internet and asynchronous communication drastically changes the 
processing of surveys. For example, Web-based and e-mail based survey techniques can greatly 
increase the diversity and saturation of survey distribution. In some situations, surveying an 
entire population is now possible rather than choosing appropriate methods to sample a group. 
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For this study, the entire population of Canadian deans and program directors of post-graduate 
medical programs - all members of the Royal College and in academic positions in faculties of 
medicine - were asked to participate. Due to the nature of this group, it could easily be assumed 
that all participants had sufficient technical acumen and resources to respond to the Web-based 
format removing an often-encountered limitation with many Internet based surveys (Dillman, 
2000). This assumption is not always possible with electronic survey yet. 

Another change to the surveys reflects the logistical and time factors involved in the 
distribution of Web-surveys. Given the simple access to on-line populations, the cost of 
distributing the surveys across the country and the cost of collation of data is minimized. Quick 
turnaround of survey release to survey findings is possible resulting in almost instantaneous 
feedback. 

Web-surveys can elicit honest feedback independent of interviewer bias. Feedback is not 
lead by the interviewer as may be the case with phone surveys. Survey respondents can be kept 
anonymous and results could be stored indiscriminately by computer. This combines the 
anonymity of traditional mail-in surveys while minimizing the participant’s effort. 

Finally, time stamps can be used to track response the receipt of responses. As responses 
can be accepted at anytime, researchers may more closely examine response patterns and 
correlate the timeliness of responses to a variety of variables such as the effect of the use of 
follow-ups and reminders. 



The Use of Follow-ups/Reminders 

A large body of research has addressed the efficacy of follow-up contacts and incentives 
for increasing response rate for mail surveys (Boser, 1996; Boser et al., 1996; Green & 
Hutchinson, 1996). Most people who answer questionnaires will do so immediately after they 
receive them (Dillman, 2000). Dillman stated that follow-ups are designed to jog memories and 
rearrange priorities rather than to overcome resistance. It is asserted in the context of this study, 
however, that it was possible that those who did not answer the survey immediately chose not to 
do so due to personal opinions regarding the questions and content. It was hypothesized that the 
follow-up e-mail might motivate these people to step forward and offer their opinions, even if 
they were negative, or if they indicated a lack of familiarity with the CanMEDS roles. To this 
point, there appears to be a paucity of research examining the change in the perspectives of the 
respondents' on the survey topic before and after a follow-up. 

The response rate was expected to match previous administered mail surveys for 
physicians however the impact of using a Web-based survey design rather than using a 
traditional mail out made response rate expectations unpredictable. In their review of research on 
surveys published from 1960-1995, Boyer and Green (1996) reported the mean response rate for 
physicians (over 1 1 studies) to be 71 .8% with a range of 59.3% to 97%. This rate was higher 
than any other group of professionals reviewed. Since the web-survey was designed to be 
accessed by respondents almost instantaneously after their release by the survey designer and 
since responses could be accepted at any time of the day or night, the response rate was expected 
to be similar or better than that of traditional mail outs. To track this response rate, responses 
were electronically monitored and tagged to track response time and to code all responses. 
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Web-based surveys have been implemented because of their ability to use advanced 
design features such as color, innovative question displays, and other features that paper surveys 
are unable to achieve (Nichols and Sedivi, 1998). The RCPSC CanMEDS survey was 
customized to personalize the forms sent to candidates based on their occupation (i.e. groups 
received slightly different versions of the survey form. Only the salutation and minor 
grammatical changes differentiated the forms). This was hoped to enhance the response rate 
however this was not assessed. 

According to Dillman (2000), since the application of a Web-based survey was chosen, a 
possible bias may be anticipated due to the limitation of respondents' access to the appropriate 
resources to receive and respond to the questionnaire in a timely fashion without the technology 
as a confound to survey response. However, based on demographic information, this bias due to 
technology access was felt to be negligible. All potential survey participants are involved in the 
education of medical specialists across Canada and therefore have the access and acumen to be 
able to answer web-based surveys. This is evidenced by all respondents being in possession of 
active e-mail accounts. 



C. Mode of Inquiry 

Prior to the survey launch, an official hard-copy letter was sent by the Royal College 
advising program directors, specialty committee chairs and post-graduate deans notifying them 
that they would receive instructions via e-mail asking them to participate in a Web-based survey 
pertaining to the Royal College's CanMEDS roles. On March 19, 2001 at 10:00 am EST, an e- 
mail was sent to 561 potential survey participants in Canada requesting their response to a survey 
which was available on the Internet. A link was provided in the e-mail that allowed the 
respondents to go directly from the e-mail to the web-based survey site. 

Data were submitted electronically and stored in an on-line database with a time/date 
stamp labeling the time the data were received by the Royal College. All members of the original 
e-mail list (regardless of whether they had already responded) were sent a thank-you/follow-up 
reminder via e-mail on March 28, 2001 at 10:00 am EST. Data collection was limited to a four- 
week period beginning at the time of the initial launch of the survey on March 19. 

The short survey consisted of four questions developed and administered based on the 
survey design principles described by Dillman (2000) to ensure clear, simple administration of 
the questionnaire. 

Questions of relevance to this paper included: 

a) a 5 point Likert-type question asking respondents to report their familiarity with the 

CanMEDS roles ranging from 'Very Good', 'Good', 'Fair', 'Poor' to 'Very Poor' 

b) an open ended questions requesting any comments the participants may have regarding 

any suggestions or examples pertaining to the CanMEDS roles. 

Patterns of response were tracked from the answers to the four questions, and from the 
time the survey data was electronically received. 
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The follow-up reminder, as recommended by Dillman (2000), consisted of a short e-mail 
sent to the entire 561 potential survey participants (see Appendix A). The message contained a 
thank-you to all participants that had submitted their responses and a positively worded reminder 
to those that had not responded at that point. The e-mail also included directions to complete the 
survey if the participant had not done so. Duplicate submissions were not accepted and were 
filtered out electronically. 

It was hypothesized that respondents who were unfamiliar with the roles being queried by 
the survey, and/or respondents not having positive comments pertaining to these roles would not 
initially submit a response. Were this the case, it could be expected that responses prompted by 
the follow-up would consist of a greater number of comments from those with low familiarity, 
and would exemplify a disproportionate number of negative comments. 

To test this hypothesis, two independent researchers categorized open-ended comments 
in terms of examples of favourable comments with regard to the new CanMEDS roles, less than 
favourable responses and recommendations for the Royal College. Only favourable and less- 
than-favourable categories are considered for this analysis. These qualitative categorizations 
were then compared to time-stamp data pre- and post-follow-up reminder. 



D. Results 

A total of 356 people responded over a time frame of four weeks resulting in a 63% 
response rate. Although this is lower than the response rate reported by Boyer and Green 
(1996), it falls within the range of the 1 1 studies they reviewed. It should be noted that Boyer 
and Green do not report the duration of the survey collection. 

Of particular note: 

• 33% (1 19) of all respondents electronically filled out and returned their survey responses 
within 24 hours of the survey launch. This constitutes of almost half of all responses; 

• 4 responses were received within the 15 minutes of the survey launch; 

• 13% (58 responses) responded within 24 hours of the follow-up e-mail. 

• As is evident in Figure 1, responses dropped over time. Responses were higher during 
the week than on week-ends. 




7 



IMPACT OF FOLLOW-UPS FOR WEB SURVEYS 7 




Date 



Figure 1 Response rate by day 



Familiarity Response Rate 

Table 1 contains the trend of the responses to the question pertaining to the level of 
familiarity with the CanMEDS roles. For each level of familiarity, the number of responses and 
the percentage of responses that level was received pre- and post- follow-up reminder are 
provided. 

Table 1 



Trend of Familiarity Responses 



Pre-Follow-up 


Post-Follow-up 


Very Good 


36 


13 


Good 


71 


58 


Fair 


65 


55 


Poor 


20 


11 


Very Poor 


12 


11 



A Pearson’s Chi-square statistic was performed comparing the number of respondents 
reporting various levels of familiarity with the CanMEDS roles pre- and post-follow-up 
reminder. No significant difference was found between the time of response and the respondents' 
familiarity with the CanMEDS roles ( X 2 = 6.9, 4 df, sig = .143) . 
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Open-Ended Response 

Table 2 contains the trend of responses to the open-ended question. Candidates were 
asked to provide comments on their overall experiences and perspectives of the CanMEDS roles. 
For each week, the numbers of favourable and less favourable comments are included as well as 
the percentage of the overall comments for that category for that week. 



Table 2 



Trend of Open-Ended Responses 



Category 


Pre-Followup 


Post-Followup 


Favourable 


19 


13 


Less than 
favourable 


21 


11 



A Chi-square analysis determined that the number of favourable and less favourable 
comments received pre- and post- follow-up was not significantly different (X 2 = 0.3, 1 df, p = 
0.61 ). 

E. Conclusions 

As a web-based survey, the RCPSC CanMEDS Implementation Survey received an 
acceptable response rate within a very short time frame. Using one follow-up contact, the 
response rate was comparable to previously published response rates for physicians. 

Using web-based technology for implementing the survey allowed researchers to easily 
monitor response patterns instantaneously allowing analysis of response patterns. All 
respondents were sent the survey at the same time and all responses were marked as per the date 
and time of receipt. From a data collection perspective, data was easily and accurately collected 
and subsequent analysis was simple to execute. The response rate and characteristics of the 
respondents could be easily monitored. 

Unique to this study, an analysis was made comparing the characteristics of familiarity 
and regard for the content with comparison to response time. Analysis would suggest that 
follow-up contacts quickly improve the response rate. 

The analysis of familiarity with the CanMEDS roles suggested that respondents did not 
change their rate of response depending on their previous knowledge of the survey content. It 
does not appear that respondents were more likely to report a less-than-favourable or favourable 
opinions after a reminder was sent out. This would suggest that follow-up contacts act as a 
method of improving the response rate without necessarily eliciting responses atypical than 
those that responded quickly and without hesitation. In the case of this study, delays cannot be 
necessarily attributed to differences in familiar or favourability to the survey content. 

Potentially, the reminder served only as the impetus to complete the survey regardless of 
familiarity or survey content. Further analysis should be undertaken to explore the 
generalizability of this phenomenon to other groups and other issues. 
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Given the efficiency and flexibility of using Web-based surveys, the implications of 
differences between traditional and Internet surveys should be considered. Furthermore, with the 
potential of tracking the relationship between response time and opinion expressed, Internet 
surveys offer a new methodology for exploring the validity of survey responses. 
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APPENDIX A 

Form e-mail letter distributed as Survey Follow-up 



Dear Dr. [Program Director, Specialty Cte Chair, PG Dean - Personalized] 

RE: The RCPSC CanMEDS Implementation Survey 

Last week, we emailed you the RCPSC CanMEDS Implementation Survey. As a key 
Canadian educator and [Postgraduate Dean or Program director or Specialty Committee Chair - 
personalized], your input is highly valued and contributes to our survey results. 

If you have already completed and returned the questionnaire to us, please accept our sincere 
thanks. 

If you have not completed our survey, we would appreciate your responses as soon as possible. 
If you have attempted to complete our survey but a network error occurred, we would appreciate 
if you could please attempt to resubmit your survey. 

This FOUR QUESTION survey should take NO LONGER THAN 5 MINUTES to complete 
and submit or online. Your answers are completely anonymous and responses will only be 
reported in summary form. 



If you HAVE NOT successfully completed the CanMEDS survey, please click on the following 
link or go to this location in your Internet Browser (Netscape or Internet Explorer) 
(HYPERLINK TO WEB-SURVEYI 



We sincerely appreciate your assistance and value your input. 

If you have any questions about this survey, please phone our offices at 1-XXX-XXX-XXX ext. 
XXX or email us via XXXXXXXX@XXXXX 

Thank-you! 

Nadia Z. Mikhael, Office of Education, Royal College of Physicians and Surgeons of Canada 

Gary Cole, Manger and Senior Research Associate, Educational Research and Development 
Unit, Royal College of Physicians and Surgeons of Canada 
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